
A Snapshot of OST Programming in Philadelphia: 

An Evaluation of Six 21 st Century Community Learning Center Grantees 


Prepared by Research for Action 
April 2013 


> 


L 

RESEARCH for ACTION 


Tracey A. Hartmann, Ph.D. 

Jian Gao, Ph.D. 
Anurag Kumar 
Kimberly Edmunds 


A Snapshot of OST Programming in Philadelphia: 

An Evaluation of Six 21 st Century Community Learning Center Grantees 


Prepared by Research for Action 


April 2013 


Authors 

Tracey A. Hartmann, Ph.D. 
Jian Gao, Ph.D. 
Anurag Kumar 
Kimberly Edmunds 


Copyright © 2013 Research for Action 


About Research for Action 


Research for Action (RFA) is a Philadelphia-based nonprofit organization. We seek to use research as 
the basis for the improvement of educational opportunities and outcomes for traditionally underserved 
students. Our work is designed to strengthen public schools and postsecondary institutions; provide 
research-based recommendations to policymakers, practitioners, and the public at the local, state, and 
national levels; and enrich the civic and community dialogue about public education. For more 
information, please visit our website at www.researchforaction.org . 


Acknowledgements 

This research has been funded through the Pennsylvania 21 st Century Community Learning Centers 
(21 st CCLC) program under Title IV, Part B of the Elementary and Secondary Education Act. We express 
our appreciation to Thomas Sheaffer of the Office of Health and Opportunity, City of Philadelphia, for 
his guidance and support in year one of this project. We would also like to thank the 21 st CCLC 
providers and their staff: Catholic Social Services, Congreso de Latinos Unidos, Education Works, The 
Netter Center for Community Partnerships, Pan American Charter School, PHMC, and the Southeast 
Delco School District for their participation in year one of this research. 

This report was greatly enhanced by the contributions of many RFA staff. We especially appreciate the 
oversight provided by Dr. Kate Shaw, Executive Director, Dr. Stephanie Levin, Associate Director, 
Alison Murawski, Communications Director, and Allison Petrosky, Communications Assistant. 


Table of Contents 

Executive Summary i 

Introduction I 

About this Report I 

I. Context: Providers and Schools 3 

II. Participant Characteristics and Level of Participation 7 

A. Student School Enrollment and Demographic Characteristics 7 

B. Participation Levels 9 

Summary I I 

III. Elements of Program Quality 1 2 

A. Alignment of Program Content to Intended Student Outcomes 1 2 

B. Elements of Program Quality: Well-Prepared Staff 1 8 

C. Elements of Program Quality: Robust School Partnerships 1 9 

IV. Student Outcomes 22 

A. School Attendance 23 

B. Literacy and Math Course Grades 25 

C. Math and Reading PSSA Scores 28 

D. Behavior 30 

V. Recommendations 34 

Recommendations for Program Providers 34 

Recommendations for Philadelphia’s OST System 34 

Appendices 36 

Appendix A. Qualitative Methodology 36 

Appendix B. Quantitative Methods 37 

Appendix C. Profile of Sites Included in the Student Outcome Analysis 41 


RESEARCH for ACTION 


A Snapshot of OST Programming in Philadelphia: 

An Evaluation of Six 2 1st Century Community Learning Center Grantees 

Prepared by Research for Action 
April 2013 

EXECUTIVE SUMMARY 

21 st Century Community Learning Center (21 st CCLC) grants are designed to support out-of-school time 
(OST) programs to provide academic support for youth attending high-poverty, under-performing 
schools. The programs also offer enrichment activities such as art and music, recreation, and career and 
technical education. Pennsylvania has funded 21 st CCLC programs since 1998, and the 2011-12 awards 
are Pennsylvania’s sixth cohort of grantees. 

Of the 18 Philadelphia organizations awarded 21 st CCLC grants in 2011-12, six organizations 
representing seven providers 1 selected Research for Action (RFA) as their local evaluator. RFA 
conducted a mixed-methods evaluation that examined student demographics, elements of program 
quality, and important student outcomes. We report aggregated findings in this report, and provide a 
set of recommendations for program improvement. 

Student Participation 

• The 21 st CCLC programs enrolled students from underperforming schools. 

• Between one third and two thirds of enrolled students qualified for Free and Reduced-Price 
Lunch. 

• Representation of students with Limited English Proficiency and Students with Disabilities was 
lower in the OST programs than in the schools that they served. 2 

• The majority of elementary school students participated in 21 st CCLC programs at a meaningful 
level (90+ days). 

The majority of middle and high school students did not attend at a meaningful level (90+ days). 


1 One grantee was an intermediary organization representing two provider organizations. 

2 High schools were the exception. High school OST programs had a higher representation of LEP students than in the schools they served. 


Elements of Program Quality 


A. Alignment of Program to Student Academic Outcomes 

• Programs aimed to impact academic achievement through homework help and project -based 
learning^ which comprised a majority of program time. 

• Programs supplemented academic support with enrichment and skill-building activities to 
engage students. 

• Programs were largely unable to provide a significant amount of one-on-one or small group 
academic support or align academic support to school day activities. 4 

B. Well-Prepared Staff 

• Staffing was a challenge for all the 21st CCLC programs largely because most staff positions were 
part-time and low-paying, and therefore providers struggled to attract highly qualified 
personnel. 

• Low salaries may continue to inhibit programs’ ability to hire certified teachers. 

• Providers were taking steps to elevate the skills and training of their staff by requiring and 
offering additional professional development, and by seeking staff with some postsecondary 
training. 

C. Robust School Partnerships 

• All but one of the 21st CCLC providers had supportive relationships with the principals of the 
schools they served. Principals supported 21st CCLC programs in recruiting students and 
provided the programs access to school space. 

• OST providers hoped to improve and deepen relationships with schools in the coming school 
year. For many, this meant opening up the lines of communication with classroom teachers to 
facilitate more connection to school day activities. 


Student Outcomes 

Research on out-of-school time (OST) programs has found that these programs can have an impact on 
school attendance; academic outcomes, including standardized test scores; and socio-emotional 
outcomes . 3 4 5 21 st CCLC programs were designed to improve student outcomes in the areas of school 
attendance, academic performance, and student behavior. Our analyses examined the potential impact 
of OST program on student academic performance and behavioral outcomes. Overall: 


3 Learning activities in which students work together to investigate relevant guiding question or problem chosen by students. PBL culminates 
in projects that require students to use and develop new skills and content knowledge. 

4 Black, A. R., Somers, M.-A., Doolittle, F., Unterman, R., and Grossman, J. B. (2009). The Evaluation of Enhanced Academic Instruction in 
After-School Programs: Final Report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U.S. Department of Education; Patricia A Lauer; Motoko Akiba; Stephanie B Wilkerson; Helen S 
Apthorp; et al. (2006). Out-of-School-Time Programs: A Meta-Analysis of Effects for At-Risk Students, Review of Educational Research, 76;n. 
275.; Moss, M., Swartz, J., Obeidallah, D., Stewart, G., Greene, D. (2001). Americorps Tutoring Outcomes Study. Abt Associates. Cambridge, 
MA. 

5 Durlack, R. & Weissberg, R. (2012?). After-school programs that follow evidence-based practices to promote social and emotional 
development are effective. Expanded Learning and After-school: Opportunities for Student Success. 

http://www.expandingleaming.org/docs/Durlak&Weissberg Final.pdf : Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School 
Programs in the 21 st Century; Their Potential and What it takes to Achieve it. Issues and Opportunities in Out of School-Time Evaluation: Issue 
10. Cambridge, MA: Harvard Family Research Project 


• After taking into account pre-existing differences between students, there is no consistent evidence 

that elementary and middle school OST participants are doing better than non-participants in the 

following areas: 

o School attendance 
o Reading and Math course grades 
o Reading and Math PSSA scores 
o Suspensions 

However, higher levels of participation are associated with better outcomes. 

• For elementary school students, higher levels of participation in the OST programs were found 

to be associated with: 

o Fewer school absences; 

o A higher likelihood of earning a C or better in reading; 
o A higher likelihood of scoring proficient or advanced on PSSA Math; and, 
o Fewer out-of-school suspensions. 

• For middle school students, higher levels of participation in the OST programs were found 
to be associated with: 

o Fewer out-of-school suspensions 

Recommendations 

For Program Providers 

• Focus on increasing participation, particularly for middle and high school students. 

• Offer more centralized planning and support for program staff, particularly in instances where 
staff do not have significant dedicated paid planning time. 

• Continue to develop relationships with school personnel, particularly teachers, to facilitate 
linking of program activities to school day activities. 

• Promote an awareness of the OST program in the school community so that schools see 
programs as a resource. 

• Prioritize professional development for staff and identify resources so that part-time front-line 
staff are able to access professional development in-house or externally. 

• Consider ways to provide more one-on-one and small group tutoring efforts. 


For Philadelphia’s OST System 


Continue to Support Citywide Professional Development of OST Staff 

• Offer professional development on increasing student participation in programs, particularly 
engaging middle and high school students. 

• Provide more professional development for OST providers on school curriculum and state 
standards. 

• Provide professional development on partnering with schools and working with classroom 
teachers. 

• Consider alternate vehicles or venues for providing professional development. 

Work with the School District of Philadelphia to Develop School-Level Support for OST-School 
Partnerships 

The District could strengthen OST programs in the following ways: 

• Encourage principals to better leverage the OST programs for academic improvement of their 
students. 

• Encourage schools to have common professional development or common planning time with 
their OST providers about the curriculum and the academic improvement needs of their students. 

• Make student data accessible to program providers in real time so that they can target academic 
interventions for program participants. 

Consider Ways in Which the City Could Support the Development of the OST Workforce 


IV 


RESEARCH for ACTION 


A Snapshot of OST Programming in Philadelphia: 

An Evaluation of Six 2 1st Century Community Learning Center Grantees 

Prepared by Research for Action 
April 2013 


Introduction 

In 2011-12, 18 organizations in Philadelphia were awarded three-year 21 st Century Community Learning 
Center (21 st CCLC) grants designed to support out-of-school time (OST) programs to provide academic 
support for youth attending high-poverty, under-performing schools. In addition to providing academic 
support, the programs offer enrichment activities such as art and music, recreation, and career and 
technical education. Pennsylvania has funded 21 st CCLC programs since 1998, and these 2011-12 awards 
are Pennsylvania’s sixth cohort of grantees. 

Of the Philadelphia organizations awarded 21 st CCLC grants, six organizations representing seven 
providers 6 selected Research for Action (RFA) as their local evaluator. As part of a citywide effort to 
create a system for OST programs, RFA agreed to aggregate the data from these local evaluations to 
provide an overview of OST programming and student outcomes in Philadelphia. While these providers 
represent a third of all Cohort 6 21 st Century grantees in Philadelphia, the analysis and results presented 
in this report provide useful insights into the potential and the needs of OST programs in Philadelphia 
and help inform the development of Philadelphia’s OST system-building efforts. 

About this Report 

This report looks at the 21 st CCLC programs evaluated by RFA. It seeks to answer the following 
questions: 

1. Who participated in the 21 st CCLC programs and how frequently did they participate? 

2. What elements of program quality were in place to achieve the intended outcomes? Specifically: 

a. What was the content of the 21 st CCLC programs and how did the content support 
program goals for academic and behavioral improvement? 

b. What were the characteristics of staff who implemented the 21 st CCLC programs and 
what types of organizational supports were available to support them? 

c. What was the reported level of partnership between 21 st CCLC programs and 
the schools their students attended? 

3. What was the relationship between program participation and academic, attendance and 
behavioral outcomes? 


6 One grantee was an intermediary organization representing two provider organizations. 


RFA addresses these research questions through a mixed-methods study that includes an aggregate 
analysis of student outcomes data, program participation data, and qualitative data from the local 
evaluations including interviews with program staff, site visits, and a review of program documents. We 
report only aggregated findings to inform broader discussions about OST programs and system- 
building efforts in Philadelphia. 

The study includes data from each of the 24 sites operated by the seven provider organizations. 
However, the analysis of student outcomes focuses on a smaller subset of nine sites which operated 
programs for the entire school year and for which we had sufficient data. The analysis of student 
outcomes considers math and reading course grades, math and reading Pennsylvania System of School 
Assessment (PSSA) scores, school attendance, and suspensions, and compares OST participants to 
other non-participating students within their schools and in relationship to their level of participation 
in the program. A more detailed description of data collected and methods can be found in the 
Appendix. 

Overall, the study finds promising evidence of the benefits of the 21 st CCLC programs in each of the 
outcome areas for elementary school students who participate more frequently. We found less evidence 
of the benefits for middle school students. However, the small sample size for middle school students 
reduced the strength of the analysis. Our analysis of 21 st CCLC program content, staffing, and school 
partnerships demonstrated some elements of quality. It also suggested ways in which programs could 
increase their impact on students’ academic outcomes. 

The report begins with descriptive information about the providers and schools and then presents 
analyses addressing each of the research questions. The sections of the report are as follows: 

I. Context: Providers and Schools 

II. Participant Characteristics and Level of Participation 

III. Elements of Quality Programming: Program Content, Staff and School Partnerships 

IV. Student Outcomes: Academics, Attendance, Behavior 
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I. Context: Providers and Schools 

21 st CCLC programs are situated in a variety of settings, and providers bring a range of organizational 
resources to their programs (see Table l). At this point in the study, we are not able to tie these 
characteristics to student outcomes, but it is important to be aware of the variety of contexts in which 
programs operate. 

A diverse array of organizations received the 2 1 st CCLC Cohort 6 grants: 

• Five of the seven 21 st CCLC providers are non-profit organizations. One provider is a charter 
school and one is a university-based center. 

• Three providers focused on specific geographic regions and were based in the regions they 
served, while four providers were not focused on particular regions. 

• Three providers had specific educational and youth development emphases. Four were multi- 
service agencies. 

• Less than half of the providers had previously received 21 st CCLC funding, and the others were 
new to 21 st CCLC funding. 

• All but one provider also received OST funding from the City of Philadelphia’s Department of 
Human Services and, thus, were linked to the City’s existing professional development supports. 

• One of the grantees was an intermediary representing two providers. Therefore, while there 
were six grantees, there were seven providers. 
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Table I. Profile of Provider Organizations 


Type of 
Organization 

Geographic Focus 

Service Focus 

Previously Funded 
by 21st CCLC 

Currently 
Receives City 
DHS Funds 

CATHOLIC SOCIAL SERVICES 

Non-Profit 

City-wide 

Multi-Service 

0 

® 

CONGRESO DE LATINOS UNIDOS 

Non-Profit 

Eastern North 
Philadelphia 

Multi-Service 

® 

Cohort 5 

® 

DIVERSIFIED COMMUNITY SERVICES (PHMC Serves As Intermediary) 

Non-Profit 

South Philadelphia 

Multi-Service 

0 

® 

EDUCATION WORKS 

Non-Profit 

City- wide 

Education and Youth 
Development 

® 

Cohort 5 

® 

METHODIST SERVICES FOR CHILDREN AND FAMILIES 


(PHMC Is Grantee Serving As Intermediary) 



Non-Profit 

City-wide 

Multi-Service 

0 

® 

NETTER CENTER FOR COMMUNITY PARTNERSHIPS OF 


THE UNIVERSITY OF PENNSYLVANIA 



University-based 

Center 

West Philadelphia 

Education and Youth 
Development 

® 

Cohort 5 

® 

PAN AMERICAN CHARTER SCHOOL 

Charter School 

City-wide 

Education 

0 

0 


received funding Os) no funding 


All providers served a diverse array of schools. 


• The 24 sites in which 21 st CCLC programs operated included 20 schools and four community 
centers, which served multiple schools. 

• Programs typically drew participants from the schools in which the programs were located. 
However, the four community centers drew from several schools in their geographical areas. For 
the purposes of this study, we asked these programs to identify one primary feeder school. 

• The schools included in this study were traditional public schools, charter schools, and Catholic 
schools. 

• Schools included elementary (K-5 and K-8), middle, and high school grades. 

• Overall, the 21 st CCLC programs served students in schools that were under-performing and had 
student populations that were predominantly low-income students of color. 7 

Table 2 below displays the number and type of schools in the study. 

Table 2. Profile of Schools Served by 2 1 st CCLC Cohort 6 grants 


ELEMENTARY (K-5) 

442 

ELEMENTARY (K-8) 

444444444444444444 18 

MIDDLE (5-8) 

442 

HIGH SCHOOL (9-12) 

442 

TOTAL 

44444444444444444444 
4444 24 


Traditional Public School Charter School Catholic School 


The 2 1 st CCLC sites were located across the city. 

Figure 1 displays the locations of the 21 st CCLC provider headquarters and the sites in which programs 
were operating. 


7 Data was not available to determine the performance of these particular Catholic schools prior to the 21 st CCLC implementation. 
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Figure I . Locations of 2 1 st CCLC Providers 



Provider 

Office 

Site 

Catholic Social Services 

☆ 

4 

Congreso de Latinos Unidos 

☆ 

4 

Education Works 

☆ 

4 

Netter Center 

☆ 

4 

PHMC 

☆ 

4 

Pan American 

☆ 

4 
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II. Participant Characteristics and Level of Participation 

21 st CCLC programs are designed to serve youth in high-poverty, under-performing schools. National 
research on participation in OST programs has found that this population of students has less access to, 
and participates less frequently in, OST programs. 8 Level of participation in OST programs is a key 
determinant of the benefits students receive from the programs. 9 In this section, we examine which 
students were participating in the 21 st CCLC programs and examine their levels of participation. 

A. Student School Enrollment and Demographic Characteristics 

Figure 2 below displays the number of students enrolled in the 21 st CCLC programs by the type of school 
they attended. 

Figure 2. Number of Participants Enrolled in 2 1 st CCLC Program by the Type of School Attended 



Note: This chart is incomplete as participation data was only available for 22 of the 24 sites. 


A total of 1,641 students were enrolled in the programs for which we had participation data (22 of 24 
sites' 0 ). More than half of these students were enrolled in traditional Philadelphia public schools and 
approximately 40% were in Catholic schools (N=648). Less than 10% of these students were enrolled in 
charter schools. 

Table 3 displays the demographic characteristics of participants from 22 of the 24 programs included in 
this study. The table reports the demographic characteristics of OST participants in comparison to the 
demographics of all other students in the same schools. * 11 


8 Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School Programs in the 21 st Century; Their Potential and What it takes to Achieve it. 
Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project 

9 Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School Programs in the 21 st Century; Their Potential and What it takes to Achieve it. 
Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project 

10 Data for one program was not available because the program didn’t start until the summer. Data for another program was not provided to 
RFA. 

11 Note: Comparison data for students attending schools considers only the primary school feeder for the program. 
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Table 3. Demographic Characteristics of 21st CCLC participants 


OST PARTICIPANTS 


NON-OST PARTICIPANTS 


in the primary feeder schools 




HIGH SCHOOL (9-12) 



FRL: 56.1% | LEP:29.3% | SWD: 13.4% 




African 

American 



Latino 



White 



Asian 



Other 



Female 


FRL: 

Free and 
Reduced 
Lunch 


LEP: 

Limited 

English 

Proficiency 


SWD: 

Student 

With 

Disability 
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Participant Demographics 

Race/Ethnicity: Overall, the programs in our study served a predominantly African-American and 
Latino student population. African-American students represented 68% of students at the elementary 
school level and 54% of students at the middle school level, while Latino students represented 40% and 
85% of middle and high school participants, respectively. 

Socioeconomic Characteristics: With one exception, the population of students served by the 21 st 
CCLC programs in our study appeared to have fewer needs than the general population at their schools. 
Specifically: 

o Sixty-four percent of non-OST elementary and middle school students qualified for FRL 
(Free and Reduced Priced Lunch), while 31-34% of OST participants qualified. 12 
o Nine to 11% of non-OST elementary and middle school participants were identified as 
LEP (Limited English Proficiency); 4-7% of OST participants were identified as LEP. 
o Eleven to 17% of elementary and middle non-OST participants were identified as SWD 
(Students with Disability); 3-5% of OST participants were SWD. 
o Fifty-six percent of high-school OST participants received FRL, while 63% of non-OST 
participants did. 

o Thirteen percent of high school OST participants had disabilities, while nearly 22% of 
non-OST students in these schools were SWD. 

However, at the high school level, OST participants were more likely to be identified as LEP than non- 
OST participants. 

Gender: Male students were slightly under-represented in OST programs as compared to the non-OST 
population in the feeder schools, particularly in middle and high schools. 

B. Participation Levels 

Research has consistently identified a relationship between a student’s level of participation in an OST 
program and the benefits received. No consistent level of overall program “dosage” has been identified 
for achieving the benefits of OST programs. However, at least two studies found academic benefits for 
students who participated in programming between 60-90 daysPs In this report, we use 30 days as the 
minimum level of participation because 30 days is the cut-off used to define a program participant for 
federal 21 st CCLC reporting requirements. We use 90 days as the threshold for the upper range because 
it matches the dosage level reported in a 2004 evaluation of 21 st CCLC programs that resulted in 
positive academic outcomes for students. In this evaluation, participants showed slightly increased 


12 Free and reduced priced lunch (FRL) is a problematic indicator of income status because families are required to submit paperwork to 
qualify for this status and can cause FRL to be under-reported. However, FRL is the only indicator of socio-economic status available through 
the School District of Philadelphia. 

'3 Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School Programs in the 21 st Century; Their Potential and What it takes to Achieve 
it. Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project; Black, A. R., 
Somers, M.-A., Doolittle, F., Unterman, R., and Grossman, J. B. (2009). The Evaluation of Enhanced Academic Instruction in After-School 
Programs: Final Report (NCEE 2009-407 7). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, United States Department of Education. 
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scores on the Iowa Test of Basic Skills (ITBS) after 60 days but a “stronger impact” on the ITBS score 
after 90 days. 14 

A Note on Program Start-Up: Because grant awards from the federal government were not 
announced until January 2012, Philadelphia’s 21 st CCLC programs operated for varying amounts of 
time during the first year of the Cohort 6 grant. Of the 24 programs, 18 operated for the entire school 
year because they were running before the funding was received. Five programs were not operational 
until the spring, which gave them little opportunity to achieve a significant threshold of program 
participation. The remaining program was not operational until the summer, and thus there is no 
available data on this program for the 2011-12 school year. Figure 3 displays the level of participation 
across all sites by grade level. 

Figure 3. Percentage Enrollment by OST Participation & Grade Level 



Participation by Grade Levels 

• More than half of elementary school students attended more than 90 days, while only a third of 
middle school students attended as frequently. 

• Sixty percent of high school participants attended fewer than 30 days. 

At least three of the programs serving middle school students were among those starting late in the 
school year; consequently, students may not have had the opportunity to attend 90 days. Both high 
school programs, however, were operating for the entire school year. 


'4 Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School Programs in the 21 st Century; Their Potential and What it takes to Achieve 
it. Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project, page 6. 


10 


Summary 

The 21 st CCLC programs enrolled students from underperforming schools and between one third to two 
thirds qualified for FRL (an indicator of poverty). However, representation of students with LEP and 
SWD was lower in the OST programs than in the schools that they served. The majority of elementary 
school students participated in 21 st CCLC programs at a meaningful level (90+ days), yet the majority of 
middle and high school students did not attend 90 or more program days. 


III. Elements of Program Quality 

21 st CCLC funding is awarded to programs with the goal of improving students’ academic outcomes 
(e.g., standardized test scores and grades), as well as students’ school attendance and behavior. 
Research identifies several aspects of high-quality, academically-focused OST programming. First, the 
quality of an afterschool program is largely determined by program content , 13 and the way in which the 
program’s planned activities are implemented on a daily basis by its staff. Second, a key factor in the 
implementation of high-quality programming is the training and expertise of the staff who deliver 
program supports . 16 Third, academically-focused OST programs like 21 st CCLC need a strong 
partnership with schools to inform the development of program content as well as other supports 
provided by staff. 1 ? In this section, we examine the extent to which these elements of quality were in 
place for the 21 st CCLC Cohort 6 programs . 18 

A. Alignment of Program Content to Intended Student Outcomes 

Research suggests that OST programs can impact academic achievement, school attendance and 
behavior when the programs: 

• Set clear and consistent goals for academic improvement ; 19 

• Provide academic support that includes: 

o A meaningful amount of support ; 20 
o Small-group or one-on-one settings ; 21 

o Alignment with the school curriculum or grade-specific academic standards ; 22 and, 
o Alignment with student progress data . 23 


'» Durlack, R. & Weissberg, R. (2012?). After-school programs that follow evidence-based practices to promote social and emotional 
development are effective. Expanded Learning and After-school: Opportunities for Student Success. 

http://www.expandingleaming.org/docs/Durlak&Weissberg Final.pdf : Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School 
Programs in the 21 st Century; Their Potential and What it takes to Achieve it. Issues and Opportunities in Out of School-Time Evaluation: Issue 
10. Cambridge, MA: Harvard Family Research Project 

16 Huang, D., Dietel, R. (2011). Making after-school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California.; Nee, J. 
(2011). Core knowledge and competencies for Afterschool and Youth development professionals. National Institute for Out-of-School Time: 
Wellesley Center for Women. ; Bodilly, Susan J. and Megan K. Beckett. Making Out-of-School-Time Matter: Evidence for an Action Agenda. 
Santa Monica, CA: RAND Corporation, 2005. http://www.rand.org/pubs/monographs/MG242. Also available in print form.; Starr, E., 
Gannett, E.,; Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School Programs in the 21 st Century; Their Potential and What it takes 
to Achieve it. Issues and Opportunities in Out of School-Time Evaluation: Issue 10. Cambridge, MA: Harvard Family Research Project 
■7 Huang, D., Dietel, R. (2011). Making after-school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California 
18 Note: We were not able to conduct an assessment of program implementation. 

79 Huang, D., Dietel, R. (2011). Making after-school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California 

20 Black, A. R., Somers, M.-A., Doolittle, F., Unterman, R., and Grossman, J. B. (2009). The Evaluation of Enhanced Academic Instruction in 
After-School Programs: Final Report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U.S. Department of Education; Patricia A Lauer; Motoko Akiba; Stephanie B Wilkerson; Helen S 
Apthorp; et al. (2006). Out-of-School-Time Programs: A Meta-Analysis of Effects for At-Risk Students, Review of Educational Research . 76m. 
275.; Moss, M., Swartz, J., Obeidallah, D., Stewart, G., Greene, D. (2001). Americorps Tutoring Outcomes Study. Abt Associates. Cambridge, 
MA. 

21 Black, A. R., Somers, M.-A., Doolittle, F., Unterman, R., and Grossman, J. B. (2009). The Evaluation of Enhanced Academic Instruction in 
After-School Programs: Final Report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U.S. Department of Education; Patricia A Lauer; Motoko Akiba; Stephanie B Wilkerson; Helen S 
Apthorp; et al. (2006). Out-of-School-Time Programs: A Meta-Analysis of Effects for At-Risk Students, Review of Educational Research . 76:0. 
275.; Moss, M., Swartz, J., Obeidallah, D., Stewart, G., Greene, D. (2001). Americorps Tutoring Outcomes Study. Abt Associates. Cambridge, 
MA. 

22 Huang, D., Dietel, R. (2011). Making after-school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California 

2 3 Beckett, M., Borman, G., Capizzano, J., Parsley, D., Ross, S., Schirm, A., & Taylor, J. (2009). Structuring out-of-school time to improve 
academic achievement: A practice guide (NCEE #2009-012). Washington, DC: National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from 
http://ies.ed.gov/ncee/wwc/publications/practiceguides. 
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• Offer enrichment activities that involve skills building and hands-on, active learning . * 2 4 

Table 4 below provides an overview of the type of program content offered at 21 of 24 sites for which 
program content data was available. 25 


Table 4: Profile of 2 1 st CCLC Program Content 


2 1 st CCLC programs operated afterschool for 2.5-3 hours each day of the week. 


Activity: Homework Help 


Description: OST staff supported students as needed in completing their daily homework. 


Daily allotted time per program 

Providers (Out of a possible 7) 

Sites (Out of a 

aossible 21 ) 
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Activity:Tutoring 


Description: Students received concentrated one-to-one support in a particular subject in which they struggled. 


Daily allotted time per program 

Providers (Out of a possible 7) 

Sites (Out of a possible 21) 
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Activity: Project-based learning (PBL) 

PS L was required for all providers except one as part of their funding from Philadelphia’s Department of Human Services 


Description: Learning activities in which students work together to investigate relevant guiding question or problem 
chosen by students. PBL culminates in projects that require students to use and develop new skills and content 
knowledge. 


Daily allotted time per program 

Providers (Out of a possible 7) 

Sites (Out of a 

aossible 21) 


uc.cn 
u Q!J 








k 

k 

k 

k 

k 

k 

k 

★ 

★ 

★ 

★ 

★ 

★ 


k 

k 

k 

k 

k 

k 

k 

MINUTES 








k 

k 

k 

k 

k 

k 



2 4 Durlack, R. & Weissberg, R. (2012?). After-school programs that follow evidence-based practices to promote social and emotional 
development are effective. Expanded Learning and After-school: Opportunities for Student Success. 

http://www.exDandingleaming.org/docs/Durlak&Weissberg Final.pdf : Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School 
Programs in the 21 st Century; Their Potential and What it takes to Achieve it. Issues and Opportunities in Out of School-Time Evaluation: Issue 
10. Cambridge, MA: Harvard Family Research Project 

2 s Note: One site was not yet up and running in the time frame of the research, though we interviewed the program coordinator about the 
school and summer planning. We were not able to interview the program coordinators for two other sites. 
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Activity: Other academic enrichment activities 


Description: Included games, individual or group activities that were separate from the PBL activities 


Daily allotted time per program 

Providers (Out of a possible 7) 

Sites (Out of a possible 21) 
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Activity: Recreation 


Description: Included sports, karate, board games and non-academic games, movies 


Daily allotted time per program 

Providers (Out of a possible 7) 

Sites (Out of a possible 21) 
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Activity: Arts 


Description: Included visual arts, theater, dance 


Daily allotted time per program 

Providers (Out of a possible 7) 

Sites (Out of a possible 21) 
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Activity: Career/college preparation 


Description: Included resume development, interview skills, SAT preparation, support for college essays or college- 
related themes in programming 


Daily allotted time per program 

Providers (Out of a possible 7) 

Sites (Out of a possible 21) 

OCCURRED ON A NON-DAILY, 
PERIODIC BASIS 

★ 

★ 
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Overview of Program Content 

Across providers, program content was similar in that: 
• All programs offered homework help. 
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• All but one provider offered project-based learning. 

• All offered enrichment activities in the areas of arts and recreation. 

Across providers, program content differed in the following ways: 

• Only one site offered one-on-one tutoring on a regular basis. 

• Approximately half (n) of the sites offered art enrichment. 

• Approximately half (n) of the sites offered sports or physical enrichment activities. 

• Four sites had college preparation or college-themed activities. 

• The amount of each activity that was offered varied by site. 

Alignment of Program Content to Student Academic Outcomes: Promising Practices 

Across the 21 st CCLC providers, programs supported academic improvement in one or more of the 

following ways detailed below. 

Academic Improvement Goals: All providers clearly viewed academic improvement as a primary 
goal of their programs. However, none viewed it as their program’s sole objective. Provider staff 
members also described broader goals, such as developing students’ social skills, socio-emotional well- 
being, and future aspirations. 

Academic Support: According to research, programs demonstrating achievement level gains in math 
and reading provide between 1.5 to 3.5 hours of academic support each week, depending on the 
structure of the support (e.g., studies of one-on-one support demonstrated impact on literacy outcomes 
with 1.5 hours of tutoring per week.). 26 As depicted in Figure 4, most providers devoted between 30-60 
minutes per day, or 2-5 hours per week, to homework help— a meaningful amount of support according 
to research. Only three programs did not provide daily academic support for all of their students. Two 
of these programs served high school students and one served middle school students. These programs 
offered homework help on an “as needed” basis. When students did not request homework help, they 
were free to join their peers in social or recreational activities taking place in the program. 

Figure 4. Amount of Homework Help Provided by Number of Sites and by Grade Level 


26 Moss, M., Swartz, J., Obeidallah, D., Stewart, G., Greene, D. f200i~). Americorps Tutoring Outcomes Study. Abt Associates. Cambridge, MA. 
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7 



Varies depending on 30 minutes or less 45 minutes per day 60 minutes per day More than 60 
student need per day minutes per day 


■ Elementary ■ Middle ■ High School 


Note: Sites that served grades K-8 are represented twice in the chart— once for middle school and once for elementary school. 


• Five elementary and six middle school programs provided homework help for 6o minutes per 
day. 

• Five elementary school programs but only two middle school programs provided homework 
help for 45 minutes per day. 

• Four elementary and five middle school programs provided homework help for 30 minutes or 
less each day. 

• The two high school programs provided homework help on an as-needed basis. One elementary 
and one middle school program had a similar approach. 

Non-Academic Support: Effective OST programs augment academic support with other types of 
enrichment and skills-building activities. 27 All but one 21 st CCLC site offered hands-on and active 
project-based learning activities for 3-5 hours each week, integrating academic skills into hands-on 
projects that students developed. 

Arts activities were offered at eleven sites. These activities focused on skill development in areas such as 
dance, visual arts, and theater and were led by teaching artists. Physical and recreational activities were 
less focused on skill development but allowed students time for positive social interactions and 
relaxation. 


2 ? Durlack, R. & Weissberg, R. (2012?). After-school programs that follow evidence-based practices to promote social and emotional 
development are effective. Expanded Learning and After-school: Opportunities for Student Success. 
http://www.expandinglearning.org/docs/Durlak&Weissberg Final.pdf ; 
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Alignment of Program Content to Student Academic Outcomes: Areas for Growth 

Across providers, program content with the ability to impact academic outcomes was limited in several 

ways. 

Most programs did not offer subject-specific academic support in a one-on-one or a small 
group setting . 28 OST programs that have demonstrated impact on academic outcomes provide one- 
on-one or small group tutoring to fewer than 9 students that is focused on particular subjects like math 
or English and provides explicit instruction in those subject areas. 

• Only one of the 21 st CCLC providers was able to offer these intensive tutoring supports— and at 
only one of its sites. This provider served as a Supplementary Educational Services (SES) 
provider and was able to offer individual tutoring to students performing below proficiency on 
the Pennsylvania System of School Assessment (PSSA) exams. 

• Two other providers recruited college students to serve as tutors and were offering one-on-one 
supports in some instances, but these supports were not widely available or consistently offered 
on a daily basis. 

• In all other sites, homework help was delivered either in a large group with staff circulating and 
providing support as needed, or students divided up into small age-level groups where staff 
could better supervise and support students’ homework completion. Staff-student ratios in most 
of these smaller group settings were between 1:10 for younger students and 1:15 for older youth. 

Alignment of program activities with the school-day curriculum happened infrequently. 

Two providers had intentions of aligning their program with the school curriculum in 2011-12, but this 
did not occur largely due to the late start-up of their programs. Two other providers intended to 
increase curricular alignment of their programs in 2012-13. Curricular alignment was less of a focus for 
the other providers. 

Providers usually did not have access to student data to target interventions. Only one 
provider reported having access to student data -report cards and attendance— that would allow them 
to monitor students’ academic progress. Without access to data, providers did not have a systematic 
way of knowing which students in their programs were struggling academically nor the specific subject 
areas in which they were struggling. 

Alignment of Program Content to Student Academic Outcomes: Summary 

The 21 st CCLC programs had the opportunity to impact academic achievement through homework help 
and project -based learning (PBL), which comprised a majority of program time. In addition, programs 
supplemented academic support with enrichment and skill-building activities that can engage students. 
However, there were important limitations to the academic support provided by programs; particularly, 


28 Black, A. R., Somers, M.-A., Doolittle, F., Unterman, R., and Grossman, J. B. (2009). The Evaluation of Enhanced Academic Instruction in 
After-School Programs: Final Report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional 
Assistance, Institute of Education Sciences, U.S. Department of Education; Patricia A Lauer; Motoko Akiba; Stephanie B Wilkerson; Helen S 
Apthorp; et al. (2006). Out-of-School-Time Programs: A Meta-Analysis of Effects for At-Risk Students, Review of Educational Research . 76 :p. 
275.; Moss, M., Swartz, J., Obeidallah, D., Stewart, G., Greene, D. f200i). Americorps Tutoring Outcomes Study. Abt Associates. Cambridge, 
MA. 
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the inability to provide more one-on-one or small group academic support. Research suggests that if 
programs are able to provide more individualized support they can increase program impact. 

B. Elements of Program Quality: Well-Prepared Staff 

Skills and expertise of OST staff play a major role in whether youth become engaged in a program and 
whether the program is beneficial to them. Research has found that: 

• Higher levels of formal education among OST staff are associated with more impactful 
programming. 2 3 9 

• Well-designed professional development may support the development of a highly-effective OST 
staff. 3 ° 

However, frontline OST positions are often entry-level, part-time and low-paying, which results in high 
turnover. A survey of Philadelphia youth workers revealed that more than half of respondents (59%) 
earned less than $20,000 per year and 38% of respondents received no benefits. 31 

Well-Prepared Staff: Promising Practices 

OST providers were attempting to elevate the skills and training of their staff in two ways: 

Providers attempted to increase the educational credentials of their staff. All providers 
were aiming to employ a majority of staff with college degrees or working towards a college degree. The 
site coordinators for four of seven providers had college degrees. 

Five providers expected staff to receive some professional development throughout the 
year. Three providers required staff to receive 15 hours annually of professional development either 
within or outside of their provider organization. Professional development was offered for all OST 
providers through Philadelphia’s OST Professional Development Project which aimed to make available 
training for OST staff on a variety of topics including project-based learning and group management. 
Two providers offered in-house professional development in addition to external trainings. Lastly, one 
provider had its own professional development division which arranged training for staff on a variety of 
topics including behavior management and teaching literacy. 

Well-Prepared Staff: Areas for Growth 

Programs encountered a number challenges that made it difficult to ensure well-prepared staff. 

Low salaries made it difficult to hire well-qualified and credentialed staff. Hiring well- 
qualified staff was the most frequently mentioned challenge for the 21 st CCLC programs. Group leader 


2 ? Bodilly, Susan J. and Megan K. Beckett. Making Out-of-School-Time Matter: Evidence for an Action Agenda. Santa Monica, CA: RAND 
Corporation, 2005. http://www.rand.org/pubs/monographs/MG242. Also available in print form. 

3 ° While research on the effectiveness of PD on improving program quality is inconsistent. Several studies have found benefits for program 
staff and programs. (New York State After-School Network (2011). After-school professional development; Resources, Outcomes and 
Considerations; Out-of-School Time Resource Center Summary of Literature: Evaluating the Impact of Teacher Trainings on OST Program 
Quality (2003). 

3 1 New York State After-School Network (2011). After-school professional development; Resources, Outcomes and Considerations; Out-of- 
School Time Resource Center Summary of Literature: Evaluating the Impact of Teacher Trainings on OST Program Quality (2003). 
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positions, and sometimes site leader positions, were part-time. Salaries for part-time group leader 
positions were reported to be $10 per hour for at least two providers. Given the low salaries, providers 
found that the pool of potential hires was limited. Only three of seven providers were able to hire a 
majority of staff that had college degrees. Compensation rates also made it difficult for providers to hire 
some certified teachers who generally demand higher hourly rates. Consequently, providers expressed 
concerns that, in some cases, staff were not well-qualified to assist students with homework and other 
academic work. 

Front-line OST staff had difficulty taking advantage of professional development (PD) 
opportunities due to their part-time positions and scheduling challenges with other jobs. 

The three providers that required or made available 15 hours of external professional development for 
their staff during the year found there to be low participation in professional development. Staff who 
worked part-time in the OST program often had other employment outside the OST program, which 
made it difficult for them to access professional development. 

Providers differed in the degree of supervisory support they provided staff for program 
planning. Most providers (4 out of 7) offered some support from the central office for the development 
of high-quality site-level activities, particularly for academically-rich PBL planning. For example, one 
provider had an education specialist who helped site leaders plan PBL lessons. One common PBL plan 
was developed for all elementary and middle school students, and lesson plans and curricular materials 
were distributed to staff at each site. It was more common for site leaders of other providers to plan 
activities with the guidance or feedback from the provider’s central office manager. Providers offered 
planning templates, organizing themes, curricular materials and feedback on lesson plans. Three 
providers allowed site and group leaders to plan PBL activities autonomously without feedback or 
guidance from the program manager. 

Our data do not allow us to conclude which approach or mix of approaches was most effective. Site 
leaders, and particularly group leaders, were not always in the best position to plan academically-rich 
PBL projects. Because providers were concerned about front-line staff members’ skill levels and degrees 
of access to professional development, and because the part-time staff received limited paid planning 
time, involving full-time program managers in the planning may help to improve programming. 

Well-Prepared Staff: Summary 

Staffing was a challenge for all the 21 st CCLC programs because most of the staff positions were part- 
time, low-paying and did not attract highly qualified personnel. However, providers were taking steps to 
elevate the skills and training of their staff both through requiring and offering additional professional 
development as well as seeking to hire staff with some postsecondary training. Nonetheless, low salaries 
may continue to hamper these programs and particularly inhibit their ability to hire certified teachers. 

C. Elements of Program Quality: Robust School Partnerships 

Programs seeking to have an impact on academic outcomes also need to develop strong, multi-leveled 
partnerships with the schools that serve their students. s 2 Specifically: 


3 2 Huang, D., Dietel, R. (2011). Making after-school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California; Beckett, 
M., Borman, G., Capizzano, J., Parsley, D., Ross, S., Schirm, A., & Taylor, J. (2009). Structuring out-of-school time to improve academic 
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• Relationships with principals provide entry to the schools for purposes of recruiting students. 
Principal support is also essential for managing the ongoing program operations, including 
access to space. 

• Relationships with teachers of program participants allow program activities to be aligned with 
school-day activities, 33 and enable program staff and teachers to work together to address the 
needs of individual students. 

School-based providers were generally able to obtain the support they needed from school principals to 
recruit students and manage the logistics of program activities. However, only a few providers had 
developed relationships with teachers for the purposes of aligning program activities to the school day 
and supporting the academic needs of individual students. 

Robust School Partnerships: Promising Practices 

The 21 st CCLC programs in our study were developing relationships with principals that would allow 
them to recruit participants and access school space. Relationships with principals were fostered by 
building on pre-existing partnerships, the alignment of principal goals with 21 st CCLC programs, and 
informal communication with principals during the school day. 

Providers obtained support from principals to recruit students to the program and to use 
school spaces. All but one of the community-based and school-based providers 3 ^ found principals 
and other school staff willing to refer students to their programs and to distribute flyers and make 
announcements about the program. In the majority of sites (19 out of 24), schools provided access to 
multiple school spaces for afterschool programming, including the cafeteria, the gymnasium, and 
classrooms. While schools worked with providers to offer adequate space for programming, at times 
schools were limited in the space they could offer due to overcrowding or the use of their facility by 
other programs. In addition, some providers reported that even when space was readily available, aging 
school facilities created programming challenges. 

In several cases, providers built relationships with principals through leveraging pre- 
existing partnerships with the schools and through establishing alignment between 
principal and OST program goals. Providers had been offering OST programs prior to 21 st CCLC 
funding in 18 of the 24 sites. In the case of two providers, the pre-existing partnerships went beyond the 
OST program and included offering supports to the school during the school day. A prior history with 
the school helped providers quickly gain the principal’s acceptance of the 21 st CCLC programs because 
providers had already established trust with the principal. In instances where providers were starting 
new partnerships, we heard that principals could be cautious and limit the scope of the program until 
they could see how the provider operated. However, continuity of relationships could still be 
undermined by principal turnover. 


achievement: A practice guide (NCEE #2009-012). Washington, DC: National Center for Education Evaluation and Regional Assistance, 
Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwc/publications/practiceguides . 

33 Huang, D., Dietel, R. (2011). Making after-school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California; Beckett, 
M., Borman, G., Capizzano, J., Parsley, D., Ross, S., Schirm, A., & Taylor, J. (2009). Structuring out-of-school time to improve academic 
achievement: A practice guide (NCEE #2009-012). Washington, DC: National Center for Education Evaluation and Regional Assistance, 
Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwc/publications/practiceguides. 

34 We only heard about one site that encountered resistance from the school principal to offering the program, although the program operated 
there for the entire school year. 
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Principals’ goals for making adequate yearly progress (AYP) based on PSSA performance were also well- 
aligned with the goals of the 21 st CCLC program, making the program an easy sell for most schools. 

School-based providers were able to communicate regularly with principals for the 
purposes of managing logistical issues. Only one provider reported having formal meetings with 
principals. But school-based site leaders, particularly those who were present during the school day, 
reported frequent informal and impromptu access to principals, as needed. The four sites located in 
community centers had greater difficulty getting access to principals and staying in regular 
communication with the schools, even to obtain routine information, such as the school calendar. 

Robust School Partnerships: Areas for Growth 

The provider-school partnerships could be strengthened in several ways, detailed below. 

In a few cases, principal and OST staff expectations differed with regard to student 
behavior and the type of activities offered in the program. In four sites, providers reported 
that principal expectations for program content and behavioral expectations varied from those of the 
OST staff. In two sites, OST program staff wanted to offer students a more relaxed environment, while 
school principals wanted school norms and expectations to be maintained. Two providers also reported 
instances in which principals were not as supportive of program activities that were not academically- 
oriented and were impatient for program activities to result in tangible academic benefits for students. 
While these challenges only occurred in a few sites, it is an issue for programs to address early in their 
partnerships with a new principal. 

The majority (19 out of 24) of sites did not have regular communication with teachers 
due to teacher misperceptions of the OST program, the size of the school, and lack of 
overlap in teacher and OST program schedules. Providers noted several challenges in building 
relationships with teachers. Some site coordinators reported that teachers assumed that OST programs 
were “babysitting.” Yet they also reported that when teachers recognized the OST program as a resource 
supporting their students’ academic achievement, they were more likely to reach out to the program. 
Site coordinators also reported that in some cases, it was difficult to effectively reach out to a significant 
number of teachers in large schools. One site coordinator reported that OST staff schedules conflicted 
with the teachers’ schedules; OST staff were starting their day as teachers were completing theirs. This 
made it difficult to develop relationships. However, in five sites OST staff reported more consistent and 
regular communication with a wider range of teachers in the school. In these instances, program staff 
capitalized on the pick-up times when staff would meet program participants with their teachers in 
their classrooms or in the school cafeteria. The site coordinators with school-day presence indicated 
that they could attend teacher grade group meetings to learn about grade-level planning and share 
information about OST activities. 

Robust School Partnerships: Summary 

All but one of the 21 st CCLC providers had supportive relationships with the schools. Principals 
supported 21 st CCLC programs in recruiting students and provided them access to school space. OST 
providers hoped to improve and deepen relationships with schools in the coming school year. For many, 
this meant opening up the lines of communication with classroom teachers. 
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IV. Student Outcomes 

Research on out-of-school time (OST) programs has found that these programs can have an impact on 
school attendance, academic outcomes including standardized test scores, and socio-emotional 
outcomes. 35 21 st CCLC programs were designed to improve student outcomes in four areas: 

• School attendance 

• Literacy and Math course grades 

• Literacy and Math PSSA scores 

• School behavior 

In this section, we describe the outcomes of the 21 st CCLC programs for each of these outcome areas for 
elementary and middle school students. The section begins with a description of our analysis. 

Student Outcome Analysis 

The analysis of student outcomes draws on a subset of the 24 21 st CCLC programs that were included in 
our study. The subset of programs used for the analysis met the following criteria: 

1) They provided a full year of programming during 2011-12 

2) Two years of data were available for the analysis 

3) They served elementary and middle school students 

A total of 9 sites, representing 5 of 7 providers, met these criteria. A profile of the program content for 
the 9 sites included in the outcomes analysis can be found in the appendix. 

The analysis examines student attendance, academic, and behavior outcomes in three ways: 

1) Comparisons of OST participant and non-participant outcomes in 2011-12; 

2) Comparative analysis between OST participants and non-participants, which controls for 
student characteristics; and, 

3) Analysis examining the relationship between student outcomes and varying participation levels, 
which controls for student characteristics. 

We also separate the analysis by elementary and middle school students. 36 More details about the 
statistical methods used in analysis, as well as limitations of the analysis, can be found in the appendix. 
While our analysis controlled for background characteristics, selection bias remains given the limited 
participant information available for this study. For example, the data on student background 
characteristics is limited to demographic information and prior academic performance. We do not know 
if OST participants or non-participants differ in other ways. We also do not have data on whether OST 


35 Durlack, R. & Weissberg, R. (2012?). After-school programs that follow evidence-based practices to promote social and emotional 
development are effective. Expanded Learning and After-school: Opportunities for Student Success. 

http://www.expandingleaming.org/docs/Durlak&Weissberg Final.pdf : Little, P., Wimer, Christopher, Weiss, H. B. (2008). After School 
Programs in the 21 st Century; Their Potential and What it takes to Achieve it. Issues and Opportunities in Out of School-Time Evaluation: Issue 
10. Cambridge, MA: Harvard Family Research Project 

3 6 Note: There are 95 participants available for our analysis at middle school level, which is relative small for some analyses. 
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participants and non-participants attended programs other than the 21 st CCLC program. The impacts of 
21 st CCLC OST programs can be underestimated if some nonparticipants actually attended other 
enrichment programs. In this section we report on the results for each of the four outcome areas— 
attendance, course grades, PSSA scores, and suspensions by grade group. 

A. School Attendance 

The analysis of student attendance examines the likelihood of OST participants missing 10 or more days 
of school in comparison to non-participants, and in relationship to their level of program participation. 

Comparison of OST Participants vs. Non-Participants 
Elementary School: 

At the elementary school level, a smaller percentage of OST participants than non- 
participants were found to have 10 or more unexcused absences. As can be seen in Figure 5 
below, this was consistent at all levels of participation. For students attending 90+ days, only 3% of 
elementary school OST participants who attended had 10 or more absences while 19% of non- 
participants had 10 or more absences. 

Middle School: 

At the middle school level, a higher percentage of OST participants than non-participants 
were found to have 10 or more absences, except in the case of participants who attended 
90+ days. As seen in the figure below, 17% of OST participants who attended 90+ days and 15% of 
non-participants had 10 or more absences. 

Figure 5. Percentage of students having 10 or more unexcused absences by school level and participation level 


Elementary (K-5) Middle (6-8) 
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Comparative Analysis of OST Participants and Non-Participants Controlling for Pre-existing 
Differences 

Importantly, OST participants and non-participants differ by their demographic and family 
characteristics, such as race/ethnicity, socioeconomic status, gender, English proficiency levels, as well 
as disability status. The observed differences in student performance may be due to these pre-existing 
differences and may not reflect the potential impact of the 21 st Century OST programs. 

To further examine the impact of 21 st Century OST program on student performance, we conducted an 
analysis intended to answer whether OST participants were more likely to miss 10 or more days of 
school than non-OST students, controlling for the characteristics of students and their families. 

Elementary and Middle School: 

After taking into account the measurable differences between students’ background 
characteristics, our analyses do not provide consistent evidence that students who 
participated in OST programs in elementary or middle school were less likely to be 
absent 10 or more days than their peers who did not participate. (Detailed results of the 
comparative analysis controlling for various student characteristics are provided in the appendix Table 
2B.). 

The Relationship between Level of Program Participation and School Attendance 
We conducted a set of analyses using program participation data to examine the relationship between 
days attending the OST program and student attendance, taking into account the differences in 
race/ethnicity, gender, English proficiency levels, FRL and disability status. 

Elementary School: 

Greater OST participation was related to better school attendance for elementary school 
students. Our analyses indicate that students who attended more days in the OST program were less 
likely to have missed 10 or more days of school than elementary school students who participated fewer 
days. 

Figure 6 below displays the results of a regression analysis showing changes in school attendance as 
program participation increases. In the figure, the solid line shows the estimated probabilities at each 
participation level and the vertical line displays 30 days of participation in the OST program. The area 
within the dashed lines represents the range in which the probability is likely to fall 19 out of 20 times. 
As the range gets wider, the uncertainty increases. 
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Figure 6. The relationship between program attendance and school attendance for elementary school students 



At the elementary school 
level, students who 
attended 30 days were 
38% less likely to have 10 
or more unexcused 
absences than students 
who only participated one 
day, and students who 
attended 90 days are 62% 
less likely to have 10 or 
more unexcused absences 
than those who attended 
30 days in the OST 
program. 


Middle School: 

No relationship was found between program participation levels and school attendance 
for middle school students. Middle school OST participants who attended the program more 
frequently did not demonstrate better school attendance than those who attended less frequently. (See 
appendix for detailed results.) 

B. Literacy and Math Course Grades 

The analysis of literacy and math outcomes investigates the likelihood of OST participants receiving a C 
or better in English and math in comparison to non-participants. 

Comparison of OST Participants vs. Non-Participants 
Elementary School: 

A higher percentage of elementary school OST participants who attended more than 30 
days earned a C or better in reading and math as compared to non-participants. As can be 

seen in Figure 7, 96% of OST participants who attended 90 or more days received a C or better in 
reading, while only 88% of non-OST participants received a C or better in reading. Similarly, 91% of 
OST participants who attended 90 or more days received at C or better in math while only 87% of non- 
OST participants received a C or better in math. 


25 


Middle School: 


Math: A higher percentage of middle school OST participants attending 90 or more days 
earned a C or better in math as compared to participants attending less frequently and 
non-particip ants . 

Reading: Higher percentages of middle school OST participants attending less than 90 
days earned a C or better in reading, as compared to non-participants and participants 
attending 90 or more days. 

As shown in Figure 7 below, 92% of middle school OST participants attending 90 or more days earned a 
C or better in math as compared to non-participants. In reading, 85% of middle school OST participants 
attending 90 or more days and non- participants earned a C or better. 

Figure 7: Course Grades: Percentage of students earning a C or above, by age-group and OST subgroup. 
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Comparative Analysis of OST Participants and non-Participants Controlling for Pre-Existing Differences 

Similar to our analysis of student attendance, the differences reported in our descriptive comparison 
above may be reflective of other differences between the OST participants and non-participants besides 
participation in the program. To address this, we conducted a second analysis that controlled for 
students’ background characteristics. 
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Elementary and Middle School: 

After controlling for background characteristics, we did not find consistent evidence 
showing that elementary or middle school OST participants were more likely than non- 
participants to earn a C or better in reading or math. The results showed a weak association 
between OST participation and elementary school students’ likelihood of receiving a C or better in 
reading. 37 However, at the middle school level the results did not provide consistent evidence showing 
that OST students were more likely to receive a C or better than non-participants. The analyses also did 
not show that OST students did better in math than non-participants for all age groups when 
controlling for background characteristics. (See appendix Table 2B.) 

Analysis Examining the Relationship Between Student Course Grades and Levels of Participation 

We conducted a third analysis that examines the relationship between program participation and 
reading and math grades, after controlling for students’ background characteristics. 

Elementary School: 

Greater OST participation was related to higher reading grades. On average, elementary 
school OST participants who attended more program days were more likely to receive a C or better in 
reading than OST participants who attended fewer days. 

Figure 8 below displays the results of a regression analysis showing changes in the likelihood of 
receiving a C or better in reading as OST program participation increased. Again, the solid line shows 
the estimated probabilities of receiving a C or better at each participation level, and the vertical line 
marks 30 days participation in the OST program. The area within the dashed lines represents the range 
in which the probability is likely to fall 19 out of 20 times. As the range gets wider, the uncertainty 
increases. 


37 (p=o.i) 
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Figure 8. The relationship between level of OST participation and elementary school students’ reading grades 



Students who attended 30 
days are 1 .4 times more 
likely to receive a “C” or 
better in reading than 
students who attended 
only one day in the 
program, and students 
who attended 90 days are 
about 2 times more likely 
to receive a “C” or better 
than those who attended 
30 days in the OST 
program. 


No relationship was found between program participation and math grades for 
elementary school students. (Detailed results are reported in the appendix Table 3B). 

Middle School: 

No relationship was found between program participation and reading or math grades 
for middle school students. (Detailed results are reported in the appendix Table 3B). 

C. Math and Reading PSSA Scores 

The analysis of PSSA scores investigates the likelihood of OST participants scoring proficient or 
advanced on the literacy or math PSSA. 

Comparison of OST Participants vs. Non-Participants 
Elementary School: 

Similar percentages of OST participants who attended the program 90 days or more and non- 
participants scored proficient or advanced on both the math and reading PSSA. However, lower 
percentages of participants who attended fewer than 90 days than non-participants were 
proficient or advanced on the math and reading PSSA. 
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Middle School: 


Higher percentages of OST participants who attended the program 90 or more days than 
non-participants and participants who attended fewer than 90 days scored proficient or 
advanced on the math and reading PSSA. 

Figure 9. PSSA: Percentage of students scoring proficient or above by age-group and OST subgroup. 
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Comparative Analysis between Participants Controlling for Pre-Existing Characteristics 
In the second comparative analysis of PSSA scores, we controlled for differences in students’ 
background characteristics to test whether the differences between OST participants and non- 
participants could be attributed to the program. 

Elementary and Middle School: 

There is no consistent evidence that program participants were more likely to score 
proficient or advanced on PSSA than non-participants at elementary or middle school 
levels. After taking into account the differences in students’ background characteristics, OST 
participants were equally likely to score proficient or advanced in math and reading as non- 
participants. (See detailed results in appendix Table 2 B). 

Analysis Examining the Relationship between School Attendance and Level of Program Participation 
The third analysis looked at the relationship between program participation and likelihood of scoring 
proficient or advanced on the math and reading PSSA, after controlling for background characteristics. 
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Elementary School: 

Greater participation in OST programming is related to PSSA proficiency. On average, 
elementary school OST participants who attend more program days are more likely to score proficient 
or advanced on the math PSSA than OST participants who attend fewer days. 

Figure 10 below displays the results of a regression analysis showing changes in the likelihood of 
scoring proficient or advanced on the Math PSSA as OST program participation increased. Again, the 
solid line shows the estimated probabilities at each participation level, and the vertical line (x=3o) 
displays 30 days participation in the OST program. The area within the dashed lines represents the 
range in which the probability is likely to fall 19 out of 20 times. As the range gets wider, the uncertainty 
increases. 

Figure 10. The relationship between OST attendance and elementary school students’ performance in PSSA math 



Students who attended 30 
days are 1.2 times more 
likely to score at 
proficient or above in 
math than students who 
attended only one day in 
the program, and students 
who attended 90 days are 
about 1 .4 times more 
likely to score at 
proficient or above than 
those who attended 30 
days in the OST program. 


Middle School: 

No relationship was found between program participation and PSSA scores for middle 
school students. (Detailed results are reported in the appendix Table 3B). 

D. Behavior 

Analysis of student behavior outcomes examined the likelihood of OST participants receiving two or 
more suspensions in comparison to non-participants, and in relationship to their level of program 
participation. 
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Comparison of OST Participants vs. Non-Participants 
Elementary and Middle School: 

At the elementary and middle school level, slightly smaller percentages of OST participants who 
attended 90 or more days had two or more suspensions as compared non-participants. However, higher 
percentages of participants who attended less than 90 days had two or more suspensions than non- 
participants (see Figure 1). 

Figure I I . Percentage of students having two or more out-of-school suspensions by grade level and days attended. 



Comparative Analysis of OST Participants and non-Participants Controlling for Pre-Existing Differences 
Elementary and Middle School: 

When controlling for background characteristics, OST participation was not related to 
likelihood of suspension at either the elementary or middle school levels. OST participants 
and non-OST participants were equally likely to have two or more suspensions when controlling for 
background characteristics and the prior year’s suspensions (see detailed results in the appendix). 

Relationship between Participation Level and the Likelihood of Receiving Two or More Suspensions 

Elementary and Middle School: 


On average, students who participate in OST programs more frequently are less likely to 
receive two or more suspensions than those participating less frequently. 
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Figure 12 below displays the results of a regression analysis showing changes in the likelihood of 
receiving two or more suspensions as program participation increased. Again, the solid line shows the 
estimated probabilities at each participation level, and the vertical line (x=3o) displays 30 days 
participation in the OST program. 

Figure 12. The relationship between OST attendance and elementary school students’ suspensions 



• Both elementary and middle school students are less likely to have 2 or more out-of-school suspensions if 
they attend more days in OST programs. 

• At the elementary school level, students who attended 30 days are 3 1% less likely to have 2 or more out-of- 
school suspensions than those who participated only one day, and students who attended 90 days are about 
53% less likely to have 2 or more out-of-school suspensions than those who attended 30 days in the OST 
program. 

• At the middle school level, students who attended 30 days were 35% less likely to have 2 or more out-of- 
school suspensions than those who participated only one day, and students who attended 90 days are about 
59% less likely to have 2 or more out-of-school suspensions than those who attended one day. 


Summary: Student Outcomes 

Our analyses examined the potential impact of OST programs on student academic performance and 
behavioral outcomes. Overall: 

• After taking into account pre-existing differences between students, there is no consistent 
evidence that elementary and middle school OST participants are doing better than non- 
participants in the following areas: 
o School Attendance 
o Reading and Math course grades 
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o Reading and Math PSSA scores 
o Suspensions 


However, higher levels of participation are associated with better outcomes: 

• For elementary school students, higher levels of participation in the OST programs were 
found to be associated with: 

o Fewer school absences; 

o A higher likelihood of earning a C or better in reading; 
o A higher likelihood of scoring proficient or advanced on PSSA Math; and, 
o Fewer out-of-school suspensions. 

• For middle school students, higher levels of participation in the OST programs were found 
to be associated with: 

o Fewer out-of-school suspensions 
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V. Recommendations 

The results of this cross-site analysis of OST programming and participation outcomes offer insights 
into the strengths and challenges of OST programs in Philadelphia, and suggest recommendations for 
providers to strengthen their individual programs as well as to improve the city’s OST system-building 
efforts. 

Recommendations for Program Providers 

• Focus on Participation. The benefits of frequent participation are significant. Programs 
should work to increase participation, particularly for middle and high school students. 

• Offer more centralized planning and support for program staff, particularly in 
instances where staff do not have significant dedicated paid planning time. 
Concerns about the skill levels of staff and the lack of paid planning time suggest that front-line 
staff need more centralized support for planning activities. 

• Continue to develop relationships with school personnel, particularly teachers. 
Schools and OST providers could work together to identity periodic common planning time for 
teachers and OST staff, and to develop communications systems such as “homework logs,” 88 
through which teachers could share information about daily homework. OST staff and schools 
could also explore common professional development that would promote alignment of 
supports for students. 

• Promote an awareness of the OST program in the school community so that 
schools see programs as a resource. OST leaders could look for avenues to promote 
teacher awareness of the program so that teachers begin to view the program as a resource. OST 
leaders could present at faculty meetings or find other venues to make teachers aware of the 
academic supports they provide. 

• Prioritize professional development for staff and identify resources so that part- 
time front-line staff are able to access professional development in-house or 
externally. Providers should support part-time staff in accessing the professional development 
that would be of greatest benefit. 

• Consider ways to provide more one-to-one tutoring efforts. Programs can have the 
greatest impact on academic outcomes when they provide regular one-to-one tutoring supports. 
Programs should consider strategies, such as recruiting high school or college student 
volunteers, to provide these supports. However, this would require programs to create an 
infrastructure to support tutors, including appropriate supervision and training. 

Recommendations for Philadelphia’s OST System 

Continue to Support Citywide Professional Development of OST Staff. 

• Offer professional development on increasing student participation in programs, 
particularly engaging middle and high school students. Middle school and high school 
students had low levels of participation in the 21 st Century OST programs and program 


3 s Huang, D., Dietel, R. (2011). Making after-school programs better. (CRESST Policy Brief). Los Angeles, CA: University of California 
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participation was related to improved outcomes for students. Programs could benefit from 
support in thinking about how to better engage middle and high school students. 

• Provide more professional development for OST providers on school curriculum 
and state standards. Even if academic supports are not central to a providers’ program 
model, homework help sessions would benefit if program staff were familiar with the topics that 
students are expected to learn. For older students, it would be beneficial for staff to learn more 
about Keystone exit exams and consider ways to support students in passing those exams. 

• Provide professional development on partnering with schools and working with 
classroom teachers. Our analyses suggest that most OST programs were not working closely 
with classroom teachers and could benefit from support in thinking about how to develop 
productive working relationships with teachers. 

• Consider alternate vehicles or venues for providing professional development: 
Part-time staff have difficulty attending centralized professional development events. City- wide 
professional development could be made available at provider organizations’ sites. Alternatively, 
professional development could be offered at different regions about the city or online. 

Work with the School District of Philadelphia to Develop School-Level Support for OST-School 
Partnerships. 

The District could strengthen OST programs in the following ways: 

• Encourage principals to better leverage the OST programs for academic 
improvement of their students. This may include: organizing more regular and formal 
meetings between OST staff and teachers to further communication about individual student 
needs; helping OST staff become familiar with the curriculum and standards; and creating 
avenues for teacher input into planning. 

• Encourage schools to have common professional development or common 
planning time with their OST providers about the curriculum and academic 
improvement needs of their students. School District staff could provide professional 
development to OST providers about the District curriculum. OST providers could provide 
training for teachers on project-based learning or STEM programming. 

• Make student data accessible to program providers in real time so that they can 
target academic interventions for program participants. OST programs could better 
target academic and behavioral interventions if they had real-time access to student 
benchmarks, report cards, daily attendance and disciplinary incidents. 

Consider Ways in Which the City Could Support the Development of the OST Workforce. Clearly 
this is one of the most significant issues facing the city OST programs. There are no easy answers but 
system level OST professionals should continue to consider levers for creating a larger pool of highly 
qualified staff for OST programs. 
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Appendices 


Appendix A. Qualitative Methodology 

Research for Action conducted one round of interviews with 21 st CCLC program directors and site 
coordinators during the spring of 2012. Researchers also collected and analyzed program documents, 
e.g., Quarterly Performance Reports (QPRs) and program schedules. 


2 1 st CCLC Provider 

Program 

Directors 

Interviewed 

Site 

Coordinators 

Interviewed 

Site Visits 

Catholic Social Services 

2 

9 

4 

Congreso de Latinos 
Unidos 

2 

3 

2 

Education Works 

3 

4 

1 

Netter Center 

2 

2 

2 

Pan American Charter 
School 

1 

1 

1 

PHMC 

5 

0 

2 

Total 

15 

19 

12 


Interviews were transcribed and data were coded using Atlas ti. Analytic memos were developed for 
each local provider covering several themes including program content, student characteristics, school 
partnerships, staffing, and use of youth development principles. Provider memos were the basis for the 
provider-level local evaluation reports. Data was then combined into three matrices (academic program 
content, staffing, and school partnerships) that reflected data across all 24 sites. Data from these 
matrices formed the basis for city-wide analytic memos on the same three topics. 
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Appendix B. Quantitative Methods 

As described in the report, RFA conducted two sets of analyses to examine the potential impacts of OST 
programming on student academic performance and behavioral outcomes. The first set of analyses 
examine whether students attending OST programs are more likely to score at proficient/ advanced 
levels on PSSA math and reading exams or to earn a letter grade “C” or better in course grades than 
non-participants. These analyses also examine whether participation in OST programs reduces 
students’ probability of having to or more unexcused absences or the likelihood of receiving 2 or more 
out-of-school suspensions. Using participant data, the second set of analyses examines the relationship 
between the number of days students participated in OST programs and observed academic and 
behavioral outcomes. All the analyses use logistic regression methods, and the variables controlled in 
the models include student demographic and family characteristics (i.e., race/ethnicity, socioeconomic 
status, gender, English proficiency levels, disability status). These characteristics are deemed important 
for student academic and behavioral performance in the literature. Table lB provides the descriptive 
statistics of the variables considered in the logistic analyses. 
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Table I B. Descriptive Statistics 


Variables considered in the 

Elementary 

(K-5th) 


Middle 
(6-8 th ) 


analyses 

Mean 

N 

Mean 

N 

Scoring proficient/advanced 
on PSSA reading(20l 1-12) 

0.36 

2532 

0.49 

1047 

Scoring proficient/advanced 
on PSSA math(20 1 1-12) 

0.43 

2554 

0.51 

1060 

Earning a C or better( >=70) 
on reading grades (20 11-12) 

0.88 

4464 

0.88 

1052 

Earning a C or better( >=70) on 
math grades (20 1 1-12) 

0.87 

4508 

0.84 

1057 

Scoring proficient/advanced on 
PSSA reading(20 1 0- 1 1) 

0.51 

1614 

0.51 

1045 

Scoring proficient/advanced on 
PSSA math(20IO-l 1) 

0.61 

1623 

0.57 

1061 

Earning a C or better) >=70) 
on reading grades (2010-1 1) 

0.82 

3541 

0.83 

1050 

Earning a C or better) >=70) on 
math grades (20 1 0- 1 1 ) 

0.85 

4451 

0.82 

1047 

Receiving 2 or more out-of-school 
suspensions (201 1-12) 

0.05 

6120 

0.10 

1157 

Having 10 or more unexcused 
absences (201 1-12) 

0.18 

5370 

0.16 

988 

Receiving 2 or more out-of-school 
suspensions (2010-11) 

0.04 

5232 

0.08 

1 101 

Having 10 or more unexcused 
absences (2010-11) 

0.23 

4482 

0.26 

932 

Race/Ethnicity=Blacl< 

0.53 

6152 

0.41 

1158 

Race/Ethnicity=Hispanic 

0.20 

6152 

0.32 

1 158 

Gender=Female 

0.48 

61 14 

0.48 

1157 

Students with Limited English 
Proficiency 

0.10 

6152 

0.1 1 

1 158 

Students with Disability 

0.10 

6152 

0.15 

1158 

Students with Free Reduced 
Lunch 

0.74 

6152 

0.75 

1 158 


Table 2B reports the estimated odds ratio of the second set of analyses using the entire population of 
students, including OST participants and non-participants in feeder schools. The predictor is the 
indicator of student participation in the OST program, and other variables controlled in the logistic 
regression include student race/ ethnicity, FRL status, ELL status, and gender. Odds Ratio (OR) is a 
measure of association between the treatment and the outcome of interest (e.g. scoring 
proficient/ advanced in PSSA, receiving a C or above), controlling for the variables that may be 
predictive of the outcome under study. Specifically, student participation in OST programs is not 
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associated with odds of outcome if OR=l; student participation in OST programs is associated with 
higher odds of outcome if 0R>1; and, student participation in OST programs is associated with lower 
odds of outcome if 0R<1. Table 2B also presents results from a logistic regression adjusted for 
propensity scores (the third and fourth rows). Propensity score is the estimated probability of a student 
participating in the OST programs given the observed background characteristics as well as the student 

TO 

historical academic and behavioral performance. Since students are not randomly assigned to OST 
programs and many characteristics that are deemed important for student academic performance and 
behavioral outcomes are not available for this study, adjusting for propensity score was employed to 
reduce estimation bias. Overall, after adjusting for the propensity scores, our analyses did provide 
consistent evidence showing that students who participated in OST programs did better than non- 
participants (OST participants are more likely to receive a C or better in reading at elementary level 
(a=0.1). 


Table 2B. Estimated odds ratio from analyses using full sample 



Scoring 
Prof./adv. in 

Scoring 
Prof./adv. in 
PSSA Math 

Receiving 
C or 

Receiving 
C or 

Receiving 2 
or more 
out-of 
school 
Suspension 

Absences 
10 or 

Adjustment 

for 


PSSA 

Reading 

better in 
Reading 

better in 
Math 

more 

days 

Propensity 

Score 

Elementary 

(k-5th) 

0.99 

0.90 

1.81** 

1 72** 

0.82 

0.30** 

No 

Middle 

(6th-8th) 

0.76 

0.54* 

1.66 

0.75 

2.10* 

2 44** 

Elementary 

(k-5th) 

1.05 

0.97 

2.72+ 

1.13 

0.74 

0.65 

Yes 

Middle 

(6th-8th) 

0.57 

0.34* 

1.31 

0.58 

1.43 

1.52 


tp<O.IO; *p< 0.05; **p<0.0l 


Table 3B reports the statistical results from logistic analyses using participant data. The predictor is 
total days attended in the OST program. These sets of analyses examine the correlations between days 
students attended the OST program and students’ academic and behavioral outcomes. After taking into 
account of student differences in race/ ethnicity, gender, English proficiency levels, as well as FRL and 
disable status, results show that attending more days in OST programs do matter for some measures of 
student academic and behavioral performance. 


39 Rosenbaum, P., & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41- 
55- 
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Table 3B. Estimated odds ratio from analyses using data of OST participants 



Scoring 
Prof./adv. in 
PSSA Reading 

Scoring 
Prof./adv. in 
PSSA Math 

Receiving a C 
or better in 
Reading 

Receiving a C 
or better in 
Math 

Receiving 2 or 
more out-of 
school 
Suspensions 

Absences 
1 0 or more 
days 

Elementary 

(k-5th) 

1.00 

1.01 + 

1 .01* 

1.00 

0.98* 

0.98** 

Middle 

(6th-8th) 

0.98 

1.01 

0.99 

1.01 

0.99* 

0.99 


'p<0. 10; *p<0.05; **p<0.0l 


While our analysis provides promising evidence showing that positive student academic and behavioral 
outcomes are associated with students’ participation levels in the 21 st Century OST programs, it is 
important to note that readers should be cautious about making causal references based on these 
results given the limitations associated with this study. One possible example of this study’s limitations 
is that, if students were not randomly assigned into the OST programs, our estimation may be biased. 
Although this study incorporated propensity score matching methods into our analysis to adjust for 
potential confounding, the number of covariates available for our analysis was relatively small. 
Furthermore, estimation can be inaccurate if students attended other enrichment programs as such 
information was not available for our analysis. 
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Appendix C. Profile of Sites Included in the Student Outcome Analysis 

Table 1C. Site Descriptions 


Site 

Provider 

Grade 

Level 

Enrolled 

Students 

Enrolled 

Morton 

Education 

Works 

K-8 

181 

Fairhill 

Congreso 

K-6 

104 

Casa Del 
Carmen 

Catholic 

Social 

Services 

K-5 

39 

NE 

Family 

Services 

Catholic 

Social 

Services 

K-6 

82 

Lea 

Netter 

Center 

1-8 

1 16 

Comegys 

Netter 

Center 

1-6 

70 

Welsh 

PHMC- 

Methodist 

K-5 

28 

Farrell 

PHMC- 

Methodist 

K-4 

39 

Girard 

PHMC- 

Diversified 

K-4 
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Table 2C. Program Characteristics 


Program Characteristics of 
Sites Included in Outcomes 
Analysis 

Number of Sites with 
Program Characteristic 

Homework help 45-60 minutes 

3 sites 

1:1 or small group tutoring 

1 site + three sites that had 
occasional opportunities for 
tutoring 

Additional enrichment activities 

All conducted project-based 
learning 


5 sites hired artists or 
contracted with art organizations 
to provide art activities 
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